User-Defined Relevance Criteria: An Exploratory Study

The objective of this study was to describe the criteria mentioned by users evaluating the information within documents as it related to the users’ information need situations. Data were collected by asking users in an academic environment to evaiuate representations and the full text of documents that had been retrieved specifically for each user’s information need situation. Users were asked to mark the portions of the document representations or of the full text of documents that indicated to the users whether they would or would not pursue the information within documents. An open-ended interview technique was then employed to discuss each marked portion with users. The interviews were audiotaped, the tapes transcribed, and the transcriptions were content analyzed in order to identify and describe evaluation criteria. The results indicate that the criteria employed by users included tangible characteristics of documents (e.g., the information content of the document, the provision of references to other sources of information), subjective qualities (e.g., agreement with the information provided by the document) and situational factors (e.g., the time constraints under which the user was working). The implications of this research for our understanding of the concept of relevance, and for the design and evaluation of information retrieval systems, are discussed.